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ABSTRACT 

Objective: Despite the increasing numbers of 
published trials of quality improvement (Ql) 
interventions in diabetes, little is known about the 
risk of bias in this literature. 
Design: Secondary analysis of a systematic review. 
Data sources: Medline, the Cochrane Effective 
Practice and Organisation of Care (EPOC) database 
(from inception to July 2010) and references of 
included studies. 

Eligibility criteria: Randomised trials assessing 11 
predefined Ql strategies or financial incentives targeting 
health systems, healthcare professionals or patients to 
improve the management of adult outpatients with 
diabetes. 

Analysis: Risk of bias (low, unclear or high) 
was assessed for the 142 trials in the review 
across nine domains using the EPOC version of the 
Cochrane Risk of Bias Tool. We used Cochran- 
Armitage tests for trends to evaluate the improvement 
over time. 

Results: There was no significant improvement over 
time in any of the risk of bias domains. Attrition 
bias (loss to follow-up) was the most common 
source of bias, with 24 trials (17%) having high risk 
of bias due to incomplete outcome data. Overall, 69 
trials (49%) had at least one domain with high risk 
of bias. Inadequate reporting frequently hampered 
the risk of bias assessment: allocation sequence was 
unclear in 82 trials (58%) and allocation 
concealment was unclear in 78 trials (55%). There 
were significant reductions neither in the 
proportions of studies at high risk of bias over 
time nor in the adequacy of reporting of risk of bias 
domains. 

Conclusions: Nearly half of the included Ql trials 
in this review were judged to have high risk of 
bias. Such trials have serious limitations that put 
the findings in question and therefore inhibit 
evidence-based Ql. There is a need to limit the 
potential for bias when conducting Ql trials and 
improve the quality of reporting of Ql trials so that 
stakeholders have adequate evidence for 
implementation. 



ARTICLE SUMMARY 



Article focus 

■ Reliable quality improvement research is needed 
to make decisions about initiating or scaling up 
quality improvement strategies. 

■ The number of published quality improvement 
trials has increased rapidly over time. 

■ The quality of trials published in other areas of 
health seem to be improving over time but the 
risk of bias in the quality improvement literature 
is uncertain. 

Key messages 

■ Nearly half of quality improvement trials for dia 
betes are at high risk of bias. 

■ The quality of quality improvement trials does 
not seem to be improving over time. 

■ Policy-makers, administrators, clinicians and 
research funders must carefully scrutinize the 
methods used in quality improvement trials to 
ensure evidence-based quality improvement. 

Strengths and limitations of this study 

■ This is the largest systematic review of risk of 
bias in the quality improvement literature and the 
only to assess for trends over time. 

■ The risk of bias tool does not capture all sources 
of methodological bias and poor reporting inter- 
feres with the assessment of many domains. 

■ The merits of any given trial report depend to 
some extent on the needs of the reader, such 
that some trials with high risk of bias may be of 
value for certain purposes. 



INTRODUCTION 

There is significant interest in quality 
improvement (Ql) in healthcare, as evi- 
denced by the rapidly increasing number of 
randomised clinical trials (RCTs) of Ql inter- 
ventions, especially in the diabetes litera- 
ture. 1 RCTs can provide a foundation for 
making statements regarding causation, but 
the validity of trials varies widely; trials with 
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adequate allocation concealment and blinding generally 
produce smaller effect sizes. 2 Since internal validity in 
QI trials is a necessary precursor for application to other 
settings, 3 the 'risk of bias' of the findings should be 
assessed to ascertain the utility of the trial results. When 
an RCT is deemed to have high risk of bias, the study's 
findings become questionable. 4 

Evaluations to assess trends in methodological quality 
of RCTs have been conducted in many fields of health- 
care, but no previous reviews have assessed risk of bias 
in QI RCTs or whether risk of bias in QI RCTs has 
changed over time. Recently, we conducted a systematic 
review and metaregression that included 142 RCTs evalu- 
ating QI strategies to improve care for patients with dia- 
betes. 1 In this secondary analysis of those data, we aimed 
to examine the risk of bias of included studies using the 
Cochrane Risk of Bias tool developed by the Cochrane 
Effective Practice and Organisation of Care (EPOC) 
group 6 and determine whether the proportion with 
high risk of bias decreased over time. We also evaluated 
the trial and publication characteristics that might be 
associated with high risk of bias. Finally, we assessed 
whether the adequacy of reporting of risk of bias 
domains improved over time. 

METHODS 

A detailed description of the methods used for search- 
ing, screening and abstracting the relevant data has 
been published 1 and is briefly summarised here. 

Search strategy 

Studies were identified by searching MEDLINE and the 
Cochrane EPOC database (up to July 2010), and screen- 
ing references of included RCTs. The search strategy has 
been previously published 1 and is available on request. 

Study selection 

RCTs examining 1 of the 11 predefined QI strategies, 
and/or financial incentives, targeting health systems 
and/or healthcare professionals for the management of 
adult outpatients with diabetes were included. RCTs had 
to report at least one of the chosen process of care mea- 
sures (proportion of patients taking acetylsalicylic acid, 
statins, antihypertensive medication, screened for retin- 
opathy, screened for foot abnormalities and monitored 
for renal function) or intermediate outcomes (glycosy- 
lated haemoglobin levels, low-density lipoprotein choles- 
terol levels, diastolic and systolic blood pressure, 
proportion of patients with controlled hypertension and 
proportion of patients who quit smoking) for inclusion. 

Data abstraction 

A draft data abstraction form was developed and modi- 
fied after a training exercise among reviewers. Two 
reviewers abstracted relevant data for each RCT inde- 
pendently. Discrepancies were resolved by discussion or 
the involvement of a third reviewer. Authors of the 



included RCTs were contacted to obtain further infor- 
mation for data items requiring clarification. Journal 
impact factors from journal citation reports (ISI Web of 
Science, 2009) were obtained. When a journal's ranking 
was unavailable, we used the impact ranking of the open 
access SMImago journal and country rank database if 
available. 7 This ranking is calculated using a similar 
formula and is strongly correlated with the journal cit- 
ation impact factor. 8 

Assessing risk of bias 

As the included trials tested QI interventions, the 
Cochrane EPOC Risk of Bias Tool 6 was used to assess 
the risk of bias in each study. The standard Cochrane 
Risk of Bias Tool includes an assessment of seven 
domains: sequence generation, allocation concealment, 
blinding of participants and personnel, blinding of 
outcome assessment, incomplete outcome data, selective 
reporting and others. The Cochrane Handbook 9 pro- 
vides instructions for making judgements about the spe- 
cific domains as high, unclear or low risk. When 
formulating summary assessments for each trial, classifi- 
cation of a study as 'high risk' indicates that bias could 
have affected the results, while unclear risk of bias indi- 
cates that some doubt exists about the results, and low 
risk of bias indicates that bias is unlikely to affect the 
results. It has been shown empirically that studies classi- 
fied as high risk using this tool are more likely to have 
larger effect sizes. 10 

The EPOC tool was adapted to account for the unique 
features of QI trials. (The guidelines for applying the 
Cochrane EPOC tool are summarised in table 1.) For 
example, in many QI trials, it is not possible to blind 
participants. In addition, QI trials may require cluster- 
randomisation to avoid contamination, but in cluster- 
randomised trials balance at baseline is a particular 
concern. 11 Therefore, the EPOC tool uses the same 
approach as the general Cochrane Risk of Bias Tool, but 
requires an assessment of bias in nine domains: 
sequence generation, allocation concealment, similarity 
of baseline measurements, similarity of baseline charac- 
teristics, incomplete data, blinding of outcome assess- 
ment, contamination, selective outcome reporting and 
others. If a given domain is deemed 'unclear', it was 
inadequately reported to determine whether it meets 
high risk or low risk criteria. Risk of bias assessment was 
conducted independently by a clinician-researcher 
(NMI) and a systematic review methodologist (ACT), 
and conflicts were resolved by discussion with an expert 
QI trialist (JMG). 

ANALYSIS 

For each risk of bias domain, the proportions of RCTs 
meeting the criteria for high or low or unclear risk of 
bias were determined. To assess for trends over time in 
the bias classifications, year of publication was cate- 
gorised into three groups demarcated by the publication 
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Table 1 Cochrane Effective Practice and Organisation of Care risk of bias assessment tool* 



Risk of bias domain 



Low risk of bias 



High risk of bias 



Unclear risk of bias 



Was the allocation 
sequence adequately 
generated? 

Was the allocation 
adequately concealed? 



Were baseline outcome 
measurements similar? 



Were baseline 
characteristics similar? 



Were incomplete 
outcome data 
adequately addressed? 
Was knowledge of the 
allocated interventions 
adequately prevented? 



Was the study free from 
selective outcome 
reporting? 
Was the study 
adequately protected 
against contamination? 

Was the study free from 
other risks of bias? 



A random component in the 
sequence generation process is 
described (eg, referring to a 
random number table) 
The unit of allocation was by 
institution, team or professional 
and allocation was performed on 
all units at the start of the study; or 
if the unit of allocation was by 
patient or episode of care and 
there was some form of 
centralised randomisation 
scheme, an on-site computer 
system or sealed opaque 
envelopes were used 
Performance or patient outcomes 
were measured prior to the 
intervention, and no important 
differences were present across 
study groups, or if imbalanced but 
appropriate adjusted analysis was 
performed 

Baseline characteristics of the 
study and control providers are 
reported and similar 

Missing outcome measures were 
unlikely to bias the results 

The authors state explicitly that 
the primary outcome variables 
were assessed blindly, or the 
outcomes are objective (eg, length 
of hospital stay) 
There is no evidence that 
outcomes were selectively 
reported 

Allocation was by community, 
institution or practice and it is 
unlikely that the control group 
received the intervention. 
There is no evidence of other risk 
of biases 



Non-random method is used 
(eg, performed by date of 
admission) 

Allocation was not adequately 
concealed 



Important differences were 
present and not adjusted for in 
analysis 



No report of characteristics in 
text or tables or if there are 
differences between control 
and intervention providers 
Missing outcome data was 
likely to bias the results 

Outcomes were not assessed 
blindly and not objective 



Some important outcomes are 
omitted from the results 

It is likely that the control group 
received the intervention. 



Not specified in the paper 



Not specified in the paper 



If no baseline measure of 
outcome 



Not clear in the paper 



Not specified in the paper 



Not specified in the paper 



Not specified in the paper 



Communication between 
intervention and control 
professionals could have 
occurred 



*Adapted for ease of presentation. See http://epoc.cochrane.org/epoc-resources-review-authors for full explanation. 



of the 2001 CONSORT statement 1 ^ and the publication 
of the earlier version of the systematic review of diabetes 
QI interventions in 2006, 13 as we believed that these may 
have spurred investigators to improve the quality of their 
trial. Therefore, we categorised the year of publication 
as before 2002; 2002-2006; and 2007-2010. We exam- 
ined each of the risk of bias domains for change over 
time descriptively and conducted either exact or asymp- 
totic Cochran-Armitage tests for trend for each item. 

We estimated the proportion of QI RCTs at high risk 
of bias overall, together with 95% asymptotic CI. For this 
analysis, we created a dichotomous indicator for each 



RCT based on whether or not the study was classified as 
high risk of bias in at least one domain. To assess for 
trends in reporting over time, we dichotomised domains 
as 'reported' (low or high risk of bias) and 'unreported' 
(unclear risk of bias). We tested for trend over time 
in the proportion at high risk of bias overall, hypothesis- 
ing that the proportion would decline over time. We 
used the same year of publication categories and con- 
ducted Cochran-Armitage tests for trend of the dichot- 
omous indicator. 

We also conducted a post hoc sensitivity analysis that 
applied an empirically based rule for assigning high risk 
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of bias overall. Since previous meta-analyses have found 
that high risk of bias in four specific domains, namely 
allocation sequence generation, allocation concealment, 
blinding and selective outcome reporting are each asso- 
ciated with greater effect size, 14-16 we repeated the ana- 
lyses considering only studies with high risk of bias in 
these domains as high risk of bias overall. 

Finally, we tested for associations between high risk of 
bias in at least one domain and study characteristics 
chosen a priori: type of diabetes (type 1, type 2, both or 
unclear), type of allocation (cluster randomised or 
patient randomised), country (USA or Canada, UK or 
Western Europe or others), type of intervention (single 
or multiface ted), journal impact factor, effective sample 
size and year of publication using % 2 tests (or Fisher's 
exact tests, as appropriate) for categorical and Wilcoxon 
signed-rank tests for continuous measures. We hypothe- 
sised that each of these characteristics may be associated 
with studies at high risk of bias overall. 

All analyses were conducted in SAS V.9.2. 17 



RESULTS 

See figure 1 for a study flow diagram. 

We analysed 142 studies, with 37 (26%) published 
before 2002, 46 (32%) between 2002 and 2006, and 59 
(42%) between 2007 and 2010. These studies evaluated 
the effects of QI interventions on 123 529 patients with 
diabetes. Trial and patient characteristics are described 
in table 2. The proportions of studies judged to be at 
low, unclear or high risk of bias for each domain are illu- 
strated in figure 2. The domains most commonly at high 
risk of bias were outcome reporting bias (17%) and simi- 
larity across characteristics at baseline (16%). A lack of 
similarity in outcome measures at baseline (10%), and 
lack of adequate blinding (8%) were also relatively 
common domains with high risk of bias. Studies were 
rarely at high risk of bias due to the allocation sequence 



N=5592 titles and abstracts 



N=3440 excluded 



N=2152 potentially relevant full- 
text articles 



N=1990 excluded full-text studies: 

1 ) Not a quality improvement 
strategy 

2) Not a randomized trial 

3) Excluded topic 

4) Not at the provider/organization 
level 

5) Not diabetes care 

6) Data not abstractable 

7) English translation unavailable 



N=142 included trials 
(plus 20 companion reports) 



Figure 1 Study flow diagram. 



Table 2 Study and patient characteristics 


Characteristic 


Result 


ratient no i s, n { /o) 




Cluster Ku I s, n (%) 


48 (33.8) 


Number of clusters, median (IQR) 


on / H O IZ~7\ 

29 (12, 57) 


Number of patients, median (IQR) 


405.3 (203, 878) 


Duration of intervention months, 


H O /O O H c o\ 

12 (8.9, 15.0) 


median (IUK) 




Mean age in years, median (IQR) 


59.4 (54.9, 62.9) 


Percentage of male, median (IQR) 


AC\ O / A H O CC 0\ 

49.8 (41 .8, 55.9) 


Type of diabetes n (%) 




Type 1 diabetes 


o /a q\ 

y (b.oj 


Type 2 diabetes 


qo /cc o\ 
oU (OD.o) 


Type 1 and 2 diabetes 


O/l /OO ON 

34 (23.9) 


Type of diabetes unclear/NR 


HO /HO A \ 

19 (13.4) 


Number of Qls per RCT median (IQR) 


2 (0, 3.5) 


Administrators of patient 




intervention(s), n (%) 




Primary care physician 


QO /OH H \ 


Nurse 


G~7 1 A "7 0\ 

b/ (4/.*:) 


Pharmacist 


HO /HO A \ 

19 (13.4) 


Dietician 


OO / H C C\ 

22 (15.5) 


Psychiatrist 


O /O H \ 

3 (2.1) 


Psychologist 


2 (1.4) 


Ophthalmologist 


2 (1.4) 


Specialist/endocrinologist 


OH 1 -A A CS\ 

21 (14.8) 


Other 


A O /O A l~ \ 

49 (34.5) 


Location of study, n (%) 




The USA 


OO / A~7 OX 

68 (47.9) 


i ne Ur\ 


H A /O C\\ 

14 (9.9) 


Canada 


11 (7.7) 


The Netherlands 


o /c a\ 
8 (O.b) 


South Korea 


"7 1 A OA 

/ (4.9) 


Australia 


6 (4.2) 


Denmark 


O \d.A ) 


Belgium 


H /O ~7\ 

1 (U.7) 


Israel 


3 (2.1) 


Spain 


3 (2.1) 


Norway 


2 (1 .4) 


France 


2 (1 .4) 


Germany 


2 (1 .4) 


Italy 


2 (1 .4) 


Switzerland 


2 (1 .4) 


unina 


O /i A\ 


Ireland 


1 (0.7) 


New Zealand 


1 (0.7) 


Thailand 


1 (0.7) 


Taiwan 


1 (0.7) 


United Arab Emirates 


1 (0.7) 


Mexico 


1 (0.7) 


All IQRs reported as the 25th and 75th percentiles, includes 
investigators and community workers. 
N, number; NA, not applicable; NR, not reported; QI, quality 
improvement; RCT, randomised clinical trial. 



generation (4%) or allocation concealment (3%), but 
these domains were often unclearly reported (57% and 
55% unclear, respectively). Selective outcome reporting 
was deemed unclear approximately 84% of the time 
because published protocols were rarely available and it 
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Figure 2 Percentage of studies 
judged to be at low, unclear or 
high risk of bias in each risk of 
bias domain. 



Adequate sequence generation? 
Adequate allocation concealment? 
Similar baseline outcome measurements? 

Similar baseline characteristics? 
Incomplete outcome data addressed? 
Knowledge of allocated intervention prevented? 

Protected against contamination? 
Free from selective outcome reporting? 

Other risks of bias? 



84 



58 


























11 



35, 



32 



i Yes (low risk of bias) Unclear ■ No (high risk of bias) 



was often plausible that many more outcomes than 
those reported were measured. Table 3 indicates a lack 
of significant trend over time in the proportion of trials 
at high risk of bias for any given domain. Examination 
of table 3 also reveals no trends over time in quality of 
reporting for any of the risk of bias domains. 

Overall, 48.6% (69/142) of the RCTs had a high risk 
of bias in at least one domain (95% CI 40.4 to 56.8). 
Figure 3 illustrates the rapid increase in number of QI 
RCTs published over time and the cumulative propor- 
tion of trials having at least one domain with high risk of 
bias up to a given year. In general, the line representing 
the proportion at high risk of bias runs parallel to the 
number of trials published, consistently accounting for 
almost half of the studies. Table 4 indicates a lack of sig- 
nificant trend over time in the proportion of trials with 
at least one domain with high risk of bias: these propor- 
tions were 46%, 44% and 54% before 2002, between 
2002 and 2006, and after 2006, respectively. Table 4 also 
demonstrates a lack of significant association between 
any of the study characteristics considered and the pres- 
ence of high risk of bias in at least one domain. 

The sensitivity analysis, restricting studies defined as 
high risk of bias overall to those with high risk of bias in 
one of four domains (allocation sequence generation, 
allocation concealment, blinding or selective outcome 
reporting) also revealed no trends over time — the pro- 
portions were 19%, 20% and 20% before 2002, between 
2002 and 2006 and after 2006, respectively (p=0.86). 

DISCUSSION 
Main findings 

Using the Cochrane EPOC Risk of Bias Tool, 6 we found 
that nearly half of RCTs focusing on diabetes had at 
least one domain at high risk of bias. The trials were 
most often at high risk of bias due to inadequate 
follow-up of participants, a lack of similarity at baseline 
across outcome measures or covariates, or inadequate 
blinding. We also noted that the majority of RCT reports 
failed to include an adequate description of the alloca- 
tion process (ie, sequence generation and allocation 
concealment were 'unclear'). To be interpreted 



appropriately, RCTs must be completely and transpar- 
ently reported. 18 19 Our findings indicate that greater 
efforts are needed to ensure both adequate reporting 
and methodological conduct of diabetes QI trials. 

We found that poor follow-up, baseline imbalances 
and blinding were the most common sources of high 
risk of bias. Although these domains may be difficult 
fully control in QI trials, methodological approaches are 
available to mitigate and/or explore such causes of risk 
of bias. For example, sensitivity analyses may be used to 
explore the risk of bias related to loss of follow-up, and 
risk of baseline imbalances in QI trials may be reduced 
through restricted randomisation techniques, especially 
when trials are cluster-randomised with relatively few 
clusters. In addition, selective outcome reporting may be 
limited if more QI trial protocols were registered. 
Finally, although blinding may be particularly difficult to 
accomplish in QI trials, this should be clearly reported; 
if outcome assessment is not blinded, risk of bias could 
still be limited by using objective outcomes. 

Comparison to literature 

A systematic review focusing on cluster randomised trials 
found minimal improvement over time in either report- 
ing or methodological conduct. 20 We found no evidence 
for a difference in the proportion of cluster-randomised 
trials at high risk of bias compared with trials in which 
individuals were allocated. However, imbalance at base- 
line was a common source of potential bias in diabetes 
QI trials, possibly owing to the inadequate use of 
restricted randomisation in cluster trials. 21 Another sys- 
tematic review included 35 studies covering a range of 
health-related fields assessing trends over time in quality 
criteria for RCTs. 5 Of these, 26 found improvement over 
time for at least one aspect of methodological quality. 
The domain most commonly noted to have improve- 
ment was allocation concealment, but the authors noted 
that this domain remained either poorly reported or 
inadequately performed in over half of the examined 
trials. We found a similarly low proportion of studies 
clearly reporting adequate allocation concealment, and 
no evidence of improvement over time. 
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Table 3 Trends over time in proportions of trials classified high, unclear or low for each risk of bias domain 



Risk of bias domain 



Pre-2002 
N=37 



2002-2006 
N=46 



2007-2010 
N=59 



P 

Value* 



P 

Valuet 



Was the allocation sequence adequately generated? 



Low 


11 


(30%) 


19 (41%) 


25 (42%) 


Unclear 


24 


(65%) 


25 (55%) 


33 (56%) 


High 


2 


(5%) 


2 (4%) 


1 (2%) 


Was the allocation adequately concealed? 










Low 


15 


(40%) 


20 (44%) 


25 (42%) 


Unclear 


21 


(57%) 


25 (54%) 


32 (54%) 


High 


1 


(3%) 


1 (2%) 


2 (4%) 


Were baseline outcomes similar? 










Low 


31 


(84%) 


39 (85%) 


44 (75%) 


Unclear 


2 


(5%) 


3 (6%) 


8 (13%) 


High 


4 


(11%) 


4 (9%) 


7 (12%) 


Were baseline characteristics similar? 










Low 


30 


(81%) 


34 (74%) 


43 (73%) 


Unclear 


3 


(8%) 


6 (13%) 


3 (5%) 


High 


4 


(11%) 


6 (13%) 


13 (22%) 


Were incomplete outcome data adequately 










addressed? 










Low 


29 


(78%) 


33 (72%) 


38 (64%) 


Unclear 


3 


(8%) 


8 (17%) 


7 (12%) 


High 


5 


(14%) 


5 (11%) 


14 (24%) 


Was knowledge of the allocated interventions 










prevented? 










Low 


32 


(87%) 


38 (83%) 


46 (78%) 


Unclear 


3 


(8%) 


5 (11%) 


7 (12%) 


High 


2 


(5%) 


3 (6%) 


6 (10%) 


Was the study protected against contamination? 










Low 


25 


(68%) 


23 (50%) 


35 (59%) 


Unclear 


10 


(27%) 


21 (46%) 


19 (32%) 


High 


2 


(5%) 


2 (4%) 


5 (9%) 


Was the study free from selective outcome 










reporting? 










Low 


3 


(8%) 


4 (9%) 


6 (10%) 


Unclear 


32 


(87%) 


37 (80%) 


50 (85%) 


High 


2 


(5%) 


5 (11%) 


3 (5%) 


Was the study free from other risks of bias? 










Low 


27 


(73%) 


30 (65%) 


39 (66%) 


Unclear 


10 


(27%) 


16 (35%) 


20 (34%) 


High 


0 




0 


0 



0.41 



1.00 



0.87 



0.16 



0.17 



0.44 



0.54 



0.84 



0.58 



0.43 



0.82 



0.20 



0.57 



0.70 



0.61 



0.78 



1.00 



0.58 



*Exact Cochran-Armitage test for high versus low or unclear risk of bias in each domain except the last domain which was analysed as low 
versus high or unclear due to absence of studies with high risk of bias. 

fExact Cochran-Armitage test for reported (high or low risk of bias) or unreported (unclear risk of bias) in each domain. 



Previous authors have noted that QI reports may not 
contain enough information to inform generalisation 
and allow for replication in different clinical settings. 22 
Standards for Quality Improvement Reporting 
(SQUIRE) guidelines suggest that investigators conduct- 
ing trials use both SQUIRE and CONSORT to inform 
their manuscripts. 19 Journal editors should enforce the 
requirements of both SQUIRE and CONSORT for QI 
RCTs, possibly by permitting detailed information to be 
posted as online appendices. Although it might seem 
onerous to force investigators to address all items in 
SQUIRE and CONSORT, the risks of poor reporting are 
substantial. Inadequate description of context could 



omit essential preconditions or important effect modi- 
fiers for a successful QI programme, while incomplete 
description of the programme itself might lead to failure 
due to partial implementation. 

Strengths and limitations 

To our knowledge, this is the largest analysis of risk of 
bias ever reported for healthcare QI RCTs and the only 
one to assess for trends over time. The findings are 
strengthened by the rigorous methods used to prepare 
the data for the systematic review. 

QI evaluations have been criticised based on numer- 
ous criteria beyond the risk of bias domains, including 
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different results. While the overall risk of bias assess- 
ment using the Cochrane Risk of Bias Tool has been 
shown to differentiate effect sizes (ie, higher risk of bias 
studies usually have larger effect sizes), 10 studies at high 
risk of bias may still offer valuable knowledge for QI 
implementers. The merit of any given report will 
depend on the needs of the reader, while the current 
analysis provides an assessment of the progress in the lit- 
erature as a whole. 

Furthermore, we acknowledge that assigning trials with 
high risk of bias in a single domain a status of high risk of 
bias overall may be arguable. Nevertheless, our sensitivity 
analysis led to the same conclusion: there has been no 
improvement over time in the proportion of trials at high 
risk of bias in this literature and no particular study char- 
acteristics were associated with high risk of bias. 

Another potential limitation stems from our analytical 
approach regarding change over time; collapsing publica- 
tion year into three timeframes ( pre-2002, 2002-2006 and 
2007-2010) and testing for trends may have limited our 
power. These timeframes were chosen a priori based on 
the publication of important documents that we thought 
might affect the conduct and reporting of these trials. We 
felt the assumption of linear change over time underlying 
the Cochran-Armitage test for trend was appropriate and 



Table 4 Association between study characteristics and risk of bias 


Characteristic 


Studies in high risk of bias in at least 
All studies (n) one domain number (%) 


p Value* 


Year of publication 






0.37 


Pre-2002 


37 


17 (46%) 




2002-2006 


46 


20 (44%) 




2007-2010 


59 


32 (54%) 




Type of diabetes 






0.11 


Type 1 


9 


3 (33%) 




Type 2 


80 


36 (45%) 




Both 


34 


16 (47%) 




Unclear 


19 


14 (74%) 




Unit of allocation 






0.24 


Patient 


94 


49 (52%) 




Cluster (eg, provider/clinic) 


48 


20 (42%) 




Country/setting 






0.62 


The USA or Canada 


79 


41 (52%) 




The UK or Western Europe 


40 


17 (43%) 




Other 


23 


11 (48%) 




Journal impact factor 






0.87 


Greater than 3 (median) 


71 


34 (47.9%) 




Less than 3 (median) 


71 


35 (49.3%) 




Effective sample size 






0.87 


Greater than 154 (median) 


71 


35 (49.3%) 




Less than 154 (median) 


71 


34 (47.9%) 




Intervention type 






0.17 


Multifaceted (featuring more than one QI strategy) 


124 


63 (51%) 




Single intervention 


18 


6 (33%) 




*Comparing proportion of studies with at least one domain at high risk of bias against studies no domains at high risk of bias. For year of 
publication, Cochran-Armitage test for trend was conducted. For other study characteristics, % 2 (or Fisher's exact) tests for categorical and 
Wilcoxon signed-rank tests for continuous variables were used. 
QI, quality improvement. 
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Figure 3 Cumulative number of diabetes quality 
improvement trial publications at high risk of bias in any 
domain, 1990-2010. 

short duration of intervention, lack of justification for 
intervention design and poor generalisability. 23 24 Some 
important components of methodological quality do not 
relate to bias (eg, reporting of a sample size calculation). 
Thus, it is possible that studies at low risk of bias have 
important flaws with respect to methodology and/or 
reporting (and vice versa), and it is possible that using 
other scales to assess study quality could have led to 



Ivers NM, Tricco AC, Taljaard M, etal. BMJ Open 2013;3:e002727. doi:1 0.1 136/bmjopen-201 3-002727 



7 



Quality of quality improvement trials over time 



in keeping with our hypotheses (eg, high and unclear risk 
of bias would decrease gradually over time, while low risk of 
bias would increase). Risk of type 2 error is tempered by 
the number of tests performed; the lack of a significant p 
value for trend for any level of risk of bias in any domain 
supports our main conclusion. Finally, this review consid- 
ered only RCTs from the diabetes literature. It would have 
been preferable to evaluate a random sample of all QI 
trials, but adequate QI electronic literature searches are yet 
to be developed. 25 

Implications 

Published trials testing QI in diabetes are frequently at 
high risk of bias, producing results that may not be rep- 
licable. Clinicians must scrutinise the internal validity of 
the results as a first step in the process of considering 
the application of clinical findings for particular 
patients. Our findings emphasise the need for policy- 
makers, managers and/ or clinical-administrators seeking 
to implement QI interventions to apply the same 
process. 3 It is likely that QI investigators publishing RCTs 
desire for their work to have a broad impact. To help 
them in accomplishing this, research funders and 
journal editors can play an important role by ensuring 
that QI trials are reported thoroughly and transparently 
and are designed in a manner that limits the potential 
for risk of bias. 
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